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DACS 

Best  Practice” 
Initiative 


DACS  Best  Practice  Initiative 


6oals: 

•  To  provide  the  Dot)  acquisition  community  with  ’'value  added" 
information  about  "Best"  practices 

-  "One-stop  shopping" 

-  Information  tailored  to  the  needs  of  individuals 

•  To  monitor  "best"  practice  implementation  within  the  Dot) 
community 

-  extend  and  expand  upon  the  research  of  Dr.  Richard  Turner, 
OSD,  relating  to  implementation  of  "best"  practices  within  the 
DoD 

-  Find  ways  to  measure/assess  the  “value  added"  by  best  practice 
implementation 

•  To  identify  and  report  on  new  or  emerging  “best"  practices 


Synopsis  of  Turner's  Research 

To  what  degree  have  existing  SIS 
projects  within  bob  adopted  best 
practices? 


Activities: 

•  Developed  and  conducted  a 

survey  to  establish  awareness  of, 
implementation,  and  perceived 
effectiveness  of  a  set  of  32  best 
practices 

-  Participants  were  military 
software  centers  of  excellence 
-covering  90%  of  acq.  programs 

-  Practice  effectiveness  evaluated 
by  a  panel  of  experts 


Some  Findings  <&  Observations: 

•  Despite  widespread  awareness,  there  is 
very  little  actual  implementation  - 
therefore  little  value  is  being  realized. 

•  Managers  are  aware  of  -  but  choose  not 
to  implement  -  BPs.  (Note  several 
barriers) 

-  Some  practices  are  considered  effective 
but  do  not  directly  impact  on  high  risk 
areas 

•  Practices  are  constantly  evolving; 
current  BP  may  not  ref lect  future  BP 

•  Practices  may  interact  signif  icantly 
with  each  other  -  crucial  to  selecting. 


Turner,  R.G.,  “Implementation  of  Best  Practices  in  U.S.  Department  of  Defense  Software-Intensive  System  Acquisitions”,  Ph.D. 
Dissertation,  George  Washington  University,  31  January  2002 


DACS  Best  Practice  (BP)  Activities 


•  Continued  Research  on  "best"  practice 

•  BP  Profiles 

-  Individual  Documents  (for  each  practice) 

•  BP  "Architecture" 

-  Describes  the  inf  luences  and  relationships  among  the  practices 

•  ON-SOIN&  Survey 

-  Extends  Dr.  Turner's  survey 

-  Addresses  awareness  and  implementation  of  BPs 

-  Collects  information  on  practice  interrelationships  and 
influences 

•  DACS  BP  Web  Site  (to  be  developed) 

-  Disseminate/Broker  BP  information  and  resources 

-  Collect,  analyze,  and  disseminate  survey  results 

-  Review  or  participate  in  discussion  forum 

-  Review  or  submit  case  studies 
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“Best"  vs.  "Gold"  Practices 


A  "Best"  Practice  (BP)  is  ... 

•A  documented  practice  aimed  at 
lowering  an  identified  risk  in  a 
system  acquisition  and  is  required 
or  recommended  by  a  bona  fide 
DoD,  industry,  or  academic  source. 
[Turner,  2002] 

•Methodologies  and  tools  that 
consistently  yield  productivity  and 
quality  results  when  implemented  in 
a  minimum  of  10  organizations  and 
50  software  projects,  and  is 
asserted  by  those  who  use  it  to 
have  been  beneficial  in  all  or  most 
of  the  projects.  [Jones,  2000] 


A  "Gold"  Practice  (OP)  is  ... 

•A  practice  that  provides  intrinsic 
value  to  an  organization  that 
develops  software  in  terms  of  cost 
savings,  product/process 
improvements,  and/or  lowering  an 
identified  risk  irrespective  of 
whether  or  not  it  has  been 
successfully  implemented  in  other 

organizations.  [DACS,  2002] 


DACS  Sold  Practices 


Related  to  Quality 

•  Use  Past  Performance 

•  Statistical  Process  Control 

•  Compile  and  Smoke  Test  Frequently 

•  Binary  Quality  Sates  at  the  Inch  Pebble 


Related  to  Risk 

•  Formal  Risk 
Management 

•  Assess  Reuse 
Risks  and  Costs 


Related  to  Technical 
Performance 


Level 

•  Model-Based  Testing 

•  Formal  Inspections 

•  Defect  Tracking  Against  Quality 
Targets 

Related  to  Project  Management 

Establish  Clear  Goals  and  Decision  Points 


Related  to  Cost 

•  Track  Earned 
Value 

•  Best  Value 
Awards 


Common  Management  and  Manufacturing  Systems 

Metrics-Based  Scheduling  and  Management 

Quantitative  Progress  Measurement 

Plan  for  Technology  Insertion 

People- A  ware  Management  Accountability 

Require  Structured  Development  Methods 
(Iterative  Processes) 

Configuration  Management 

Program  Wide  Visibility  of  Progress  vs..  Plan 

Develop  and  Maintain  a  Life-Cycle  Business  Case 


•  Agreement  on  Interfaces 

•  Ensure  Interoperability 

•  Leverage  COTS/NDI 

•  Demonstration-Based 
Reviews 

•  Independent  Expert 
Reviews 

Related  to  Requirements 

•  Performance  Based  Specif ications 

•  Manage  Requirements 

•  Commercial  Specifications  <& 
Standards/  Open  Systems 

•  Requirements  Trade-Off/Negotiation 

Related  to  Processes 

•  Architecture-First  Approach 

•  Integrated  Product  and  Process 
Development  (IPPD) 

•  Acquisition  Process  Improvement 

•  Goal-Question-Metric  Approach 

•  Capture  Artifacts  in  Rigorous  Model- 
Based  Notation 
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C  :.III  i  F -1  i;:l  ■  :n  ■  il  il  ins  .Ti  l  :3 1 :.ar ■  i-n - . T  .-'■T' i: i/i  Sy.1- 
Common  Management  anc  MTnifacturing  Systems 

Comple  erdimoF-e  le=tl  requenttv 

Coni  g  urati  on  Malta  gemeri 

Da  lei  i  Ti  jrWt  g  Agam-'-tCual^  Gin--. 

DorrcnstiaSon-Ba»d  F  ev line  E  ■  00  tattle  Ard-iteoure; 

DeveioprtVlaimaina  -iie-Cyile  Business  Case 

Ensue  Intevoperabilty 

Estatllsti  CFe  ar  Goa  sand  Deem  on  Pams 

f  irnaihsf^ragfi* 

F  jim jl  FJisli  M;r,>geiT>=,"l; 

&oai-QiB5lon-Metnr  ADrfoeoli 

hdependeit  E>pa1  RhhsmsHSCEb 

i  leg  ated  Product  and  P rote ssOe-^e loe noert 

L  ev ) :  i  i  r.  •  ■  C  OT  STJj  1 

Manage  Requ  e™_its 

t/etr  :i  Ba»c  Echedd  >□  an  t  h^nagement 

t/odel-Eased  leshng 

People- A.-,  ire  M  inagerneiit  Accoumabii^ 

Ferieimijnw-E;  r.,-.ir:  Sf  O  0*0 1*015 

FianigrTs^tTrcifUH'  nsertion 
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QjanDtaoee  Progress  Me aau tame m 
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L£t  Fast  Parra  mance 
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" Architecture-First  Approach  "  Profile  Survey  Form  (Part  3  of  3-Part  Survey) 


PRACTICE 

RISK  CATEGORIES 

High 

Medium 

Low 

SE'  PR  RQ’ 

ES 

PE  1ST 

1 

WE 

MN 

QA 

CN 

Architecture -First  Approach  | 

Enter  "D"  if  Practice  has  direct  impact  on  risk  category 
Enter  "I"  if  Practice  has  indirect  impact  on  risk  category 
Leave  blank  if  Practice  has  no/negligible  impact  on  risk  category 

The  Risk  Categories  considered  in  this  part  of  the  survey  are: 

SE:  System  Engineering 
PR:  Process 

RQ:  Requirements  Quality/Stability 
ES:  Estimation 
PE:  Policy/External 
ST:  Staffing 

WE:  Working  Environment 
MN:  Monitoring 
QA:  Product  Quality 
CN:  Contracting 
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Section 


JBI 

Program 

At 

AFRL 


What  is  the  problem? 


•  Too  much  information  -from  multiple 
sources/sensors  and  residing  across  a 
multitude  of  systems 

•  Current  C2ISR  tools  only  get  us  partway 
there 

-  Large  monolithic,  rigid  enterprises 

-  Unique  information  infrastructures 

-  Information  interoperability  issues 

-  System  admin  &  conf  iguration  overhead 

•  Decision-maker  must  filter  <&  aggregate 

•  Interf  aces  between  systems  and  & 
brand  new  enterprise  systems  cost- 
prohibitive  (time  <&  $$) 

•  Results  from  the  Kosovo  experience: 

-  “Info  fatigue" 

-  “Cyber-  rubbernecking" 


Opportunity! 
Leverage  on  commercial 
IT  investment 

•  Commercial  IT 
advancing  at  a  staggering 
pace 

•  Commercial  IT 
Enterprises  face  the 
same  dilemma 
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Is  there  a  solution? 


JBI  Goals 

Increase  affordability  and 
flexibility  of  future 
information  systems 
supporting  the  war  fighter 

Provide  an  open(standards- 
based)  and  extensible 
infrastructure  upon  which 
legacy,  evolving,  and 
future  information 
systems  will  operate 


&  Challenges 

•  Achieve  universality 

•  Become  technology 
agnostic 

•  Achieve  legacy  client 
integration 

•  Embrace  and  manage  many 
domains 

•  Achieve  scalability 

•  Create  a  technical 
architecture  that  does  not 
constrain  the  solution 
space 
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What  is  JBI? 


•  A  combat  information 
management  system 
which  provides  users  with 
specific  information 
required  to  perform  their 
functional  responsibilities 
during  crisis  or  conflict. 
[SAB  report  1999] 


□□□ 


•  A  system  of  systems  that 

-  Integrates,  aggregates,  and 
distributes  information 

-  To  users  at  all  echelons  - 
from  the  command  center  to 
the  battlefield 


•Reference  "Information  Management  to  Support  the  Warrior"  (1998),  and  "Building 
the  Joint  Battlespace  Infosphere"  (1999)  published  by  the  Air  Force  Scientific  Advisory 
Board 
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JBI  Benefits 


■Impraupd  trnhnlnD  □faualablnlHRmuniii: 

■  Dvnamln  islInrlnQ  □Tlntmni'lnn  h  UtarlQh'br 

■  Gulnhpr  nDllmiDn  of  ml  cdnn  □rllral  pi.imtr; 

■  Fa  nrbr.'mm  annurafe  TOT  I D  and  pm  rzaulnn 


JBI  Impact  on  the  Battlespace 


V.V.’.V 


CAOC 


■ 

Publish/ 

Subscribe/ 

Query 

Force 

Templates 


Fuselets 


Distributed 

Collaboration 
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What  does  a  JBI  look  l  ike? 


+ir* 


c 


&rH 


Appl  i  cations , 
Systems,  Sensors 
(JBI  Clients) 


JE3  Buburlp-lnn 
Era  Her 


JH 

PJtsnaQpmpni 
BpitjIdpg 


ACCESS 


JBI  Platform  (Core  Seruces) 


Global  Grid,  SI  PR  NET,  Internet, 


i: 


Publish,  Subscribe  &  Query 

“Foundation  of  the  JBI” 


Clients  publish  information  objects: 
object  type,  metadata  &  and 
data(payload) 


Clients  subscribe  to  information  - 
look  forward  in  time  for  objects 
(Give  me  all  objects  of  type  “A” 
from  source  K  with  attributes  “m’V’n” 
&  “s”  -  as  they  are  published  ) 


Query  looks  backward  in  time  over 
the  JBI  repository  (of  objects) 


Publish 


Info 

Objects 


a 


■ 

Client 

f 

#  3 

f 

/  7 

LA 


Client 

#  4 
onent 

#  5 


Q 


/I 


Client 
#  N 


Publish  <&  cj? 
Subscribe 
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•sy 


7 _ 

7 

Client 

#  2 

f 
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Fuselets 

"Tailoring  the  Information  Space" 


•  Fuselets  are 
"Special"  JBI 
clients 

•  Publish  new  info 
object  by  refining 
or  fusing  other 
information 
objects 

•  Transforming  data 
into  knowledge 


Info 

Objects 


%/. 


'*4 


a 
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Force  Templates 

"Plugging  into  the  JBI" 


Control  entities  that 
allow  clients  (at 
varying  levels)  to 
register/identify 
themselves  to  the  JBI. 

Provide  a  mechanism 
for  seamlessly 
integrating  diverse 
coalition  forces  into 
these  new  information 
systems 

Enable  new  clients  to 
come  and  go  without 
modification  of  the 
JBI  infrastructure 


Force 

Template 


Distributed  Collaboration 


•  Use  of  shared  updateable  knowledge 
objects 

•  Collaborative  planning 

-  "Shared  whiteboard" 

-  Multiple  users  interact  with  an 
application,  see  changes  made  by  other 
users,  and  ultimately  come  to  a  common 
agreement/conclusion 
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JBI  Program  Profile 


(6.2)  Govt.  Salaries  $  2  M  + 

(6.3)  Contractors  A  Other  $  3  M  = 

Estimated  Avg  Annual  Funding  $  5  M 

JBI  Team 

Contracting  Vehicles 

Govt.  Contractors 

Military  -  4  In_House  12-15 

Civilian  -  8  (10  Companies) 

Other  (est.  12  Companies) 

•  IAC  TATs  •  BAAs 

•  SBIRs  •  PRDAs 

•  TOAs  •  Other  ... 

Collaborators  (Several  Orgs  &  Individuals') 

Cornell  Univ.  - 

Information  Assurance  Institute 

DARPA 

Other  AFRL  Croups 
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Program  Management  Activities 


Requirements 

The  vision  (and  operational  concepts) 
presented  by  Air  Force  Science  Advisory 
Board  is  driving  program  activity  -  serving  as 
the  requirements  guide. 


development 

Implementing  iterative  (spiral) 
development  process 


deliverables 


Roadmaps  identify/schedule  the 
tasks 

-Each  planned  increment 
(phase)  represents  an 
increasing  level  of  capability 


Outcomes/products  of  each  task  or  phase  are  typically 
documents  that  serve  as  requirements  for  future  efforts 
resulting  in  technology  transition. 
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Section 


“Best  Practices” 
on 
JBI 


Information  Gathering 
Approach  for  JBI  Program 

•  Conducted  interviews  with  AFRL  leaders,  and  in-house  contractor 
technical  people 

-  What  are  you  doing  ? 

-  Why  are  you  doing  it  ? 

-  How  are  you  doing  it? 

-  What  are  the  biggest  challenges?  Issues?  Successes? 

•  Answers  to  those  questions  revealed  evidence  of  certain  practices 

•  Followed  with  a  series  of  questions  designed  to  establish  qualitative 
and  quantitative  data  to  support  the  degree  of  implementation  of 
the  practices. 

•  In  parallel,  gathered  information  from  the  JBI  website 
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Awareness  of  "Go\d  Practices" 


•  JBI  team  is  not  cognizant  of  their  activities  as  exemplifying  "best 
practice". 

•  Recognize  the  intrinsic  value  ("Gold")  of  their  practices  to  achieving 
the  mission. 

-  “We  have  to  use  the  spiral(iterative)  development  process  -  there  are 
too  many  unknowns".  [Tech  Director] 

-  “Achieving  interoperability  is  a  principle  requirement  of  the  JBI  -  our 
main  focus  -  not  just  something  we  try  to  do." 

-  “To  keep  the  cost  down  we  have  to  achieve  universality  -  and  to  do  that 
we  have  to  take  the  open  systems  approach." 

•  No  formal  plan  for  assessing  the  value  of  implemented  practices 
-process  improvement  is  considered  important  -  but  addressed 
informally. 

•  RAD  "mindset"  contributes  to  a  lack  of  quantitative  data  to 
provide  objective  evidence  of  the  "success"  of  these  practices. 


DACS  Sold  Practices 


Implemented  in 
JBI! 

•  Program  Wide  Visibility  of 
Progress  vs..  Plan 

•  Agreement  on  Interfaces 

•  Architecture-First  Approach 

•  Ensure  Interoperability 

•  Commercial  Specif  ications  & 
Standards/  Open  Systems 

•  Configuration  Management 

•  Leverage  COTS/NDI 

•  Require  Structured 
Development  Methods 
(Iterative  Processes) 

•  Plan  for  Technology  Insertion 

•  Demonstration-Based  Reviews 


•  Binary  Quality  Sates  at  the 
Inch  Pebble  Level 

•  Track  Earned  Value 

•  Manage  Requirements 

•  Formal  Risk  Management 

•  Formal  Inspections 

•  Metrics-Based  Scheduling  and 
Management 

•  Defect  Tracking  Against 
Quality  Targets 

•  Quantitative  Progress 
Measurement 


Noticeably 

Absent! 


Program-Wide  Visibility  of 
Progress  vs.  Plan 

...  the  practice  of  sharing  core  indicators  of  project  health 
(or  dysfunction)  with  all  project  participants 


Weekly  meeting  of  entire  AFRL  JBI  team 

-  Well  attended  -perceived  as  worthwhile  by 
some  developers 

-  Project/task  status  reported 

-  Issues  discussed  openly 


Core 

Indicators 

Of 

Project 

Health? 


Principle  Investigators  Conference  (Spring  <&  Fall) 
-  Formal  JBI  status  review 


Architecture -First  Approach 


The  practice  of  seeking  a  demonstrable  balance  among 
driving  requirements,  architecturally  significant  design 
decisions,  and  the  life-cycle  plans  to  develop  an  architecture 
before  resources  are  committed  for  full-scale  development. 


Using  skilled  architects 
Considering  alternative  designs 

Solicited  architectural  ideas  from 
the  technical  community  (Y-JBIs) 

Leveraging  commercial 
middleware 

Using  Zachman  framework  for 
architecture  representations 

Architecture  is  evolving 

Have  initial  release  of  a  JBI 
architecture  available  for  review 
by  interested  parties 

Challenge  of  interoperability 

remains  JBI  is  architecture! 


Elicit 

Architectural 

Requirements 


t 


Design  the 
Architecture 


Document  the 
Architecture 


"  Analyze  the 
Architecture 


Iterative 

Process 


Realize  the 
Architecture 


Maintain  the 
Architecture 
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Ensure  Interoperability 

Ensuring  the  ability  of  systems,  units,  or  forces  to  provide  services 
to  and  accept  services  from  other  systems,  units,  or  forces  and  to 
use  the  services  so  exchanged  to  enable  them  to  operate  effectively 
together. 


Interoperability  is  a  primary  goal  of  JBI  -  and  the  primary 
challenge. 

Achieved  at  the  architecture  level  (Architecture  must 
demonstrate  interoperability) 

Established  an  in-house  test  cell  for  the  purpose  of  What^®9ree 
evaluating  prototypes  with  respect  to  issues  of  interoperability 
interoperability. 

-  Comprised  of  govt,  and  in-house  contractors 

-  Independent  from  contractors  doing  development 


IS 

acceptable? 
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Commercial  Specifications  & 
Standards/  Open  Systems 

The  practice  of  developing  a  technical  and  business  strategy 
3  for  software  intensive  systems  that  defines  key  interfaces  by 
widely-used  consensus -based  standards.  Standards  are 
selected  based  on  maturity,  market  acceptance,  and  allowance 
for  future  technology  insertion 


•  Standards-based  development  -  not  standardization 

•  Just  like  the  plug  that  goes  into  the  outlet  JBI  clients  must 
conform  to  specs  in  order  to  "connect"  to  JBI 

•  Now  have  a  spec  for  the  common  API  (JBI  platform) 

-  Using  JBOSS,  JMS,  ORACLE  REPOSITORIES 


Process  for 
Selecting 
Standards? 
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Configuration  Management 

The  discipline  of  identifying  the  configuration  of  a 
hardware/software  system  at  discrete  points  in  time  with  the 
purpose  of  systematically  controlling  changes  to  the  configuration 
and  maintaining  the  integrity  and  traceability  of  the  configuration 
throughout  the  system  lifecycle. 


•  Developers  of  the  common  API  (JBI  platform)  are  using  CVS, 
an  open  source  configuration  management  system,  for 
tracking  the  source  code  used  in  each  of  alternate  versions 
of  the  prototypes  under  development. 

•  CM  policy  is  communicated  verbally  to  new  developers.  No 
formalized  CM  plan. 

•  Developers  view  CM  as  "annoying,  but  necessary"  to  support 
the  mission. 
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Leverage  COTS/NDI 

The  practice  of  identifying/using  Commercial  Off-The-Shelf 
software, and/  or  Non-Development  Items  in  lieu  of  custom- 
developed  components  in  order  to  reduce  costs  and/or  improve 
quality  over  the  product  life  cycle. 


•  Developing  architecture  for  COTS  middleware 

•  JBI  tasks  identify/explore  commercial/NDI  technology 


-  Information  Objects: 

•  XML,  X  technologies 

•  Semantic  Web  :  RDF,  DAML  +  OIL 

-  Pub/Sub/Query: 

•  IBM  MQ  Series 

•  Tibco  Rendevous 

•  Talarian  Systems 

-  Fuselets:  Computer  Associates'  “Neugents" 

-  Force  Templates: 

•  Texar  Secure  Realms 

•  Oracle  Internet  Directory 

•  Netscape  iPlanet 


How  do  these 
candidate  solutions 
impact 

interoperability 

goals? 
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Plan  for  Technology  Insertion 


Planning  how  to  take  advantage  of  future  technology 
opportunities  to  improve  the  performance  or  reduce  the  cost 
of  the  system  by  replacing  existing  system  components  with 
newer  technology  components  as  they  become  available. 


•  The  design  of  JBI  is  itself  a  plan  for  technology  insertion. 

•  Milestones  for  insertion 

•  Challenge  is  to  ensure  technology  insertion  while  optimizing 
use  of  COTS,  and  without  sacrificing  interoperability. 

•  Implementing  "plug  -n-play" 

How  do  we 
validate  the 
technology 
insertion 
capability? 
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Demonstration -based  Reviews 

...  the  practice  of  using  executable  demonstrations  of 
relevant  scenarios  as  an  integral  part  of  project  reviews  to 
stimulate  earlier  convergence  on  integration,  support 
tangible  understanding  of  design  trade-offs,  and  eliminate 
architectural  defects  as  early  as  possible 


•  Demonstration  is  the  primary  review  method  for  most  tasks 
on  the  JBI  at  all  levels. 

•  Formal  demonstrations  are  project/phase  milestones 

•  Demonstrations  serve  as  gates(decision  points)  for  further 
action  and  funding 


h-u 


Section 


IV 


Ending 

Remarks 


GP  Implementation  on  JBI 

Focus  is  on  the  mission  -  not  on  process  improvement. 

Assessment  of  CP  implementation  on  JBI  triggers  many  questions: 

•  What  degree  of  implementation  is  necessary  in  order  to 
claim  that  the  practice  has  been  implemented? 

•  Can  we  (should  we)  attempt  to  refine,  and  perhaps 
standardize  the  definitions  of  CPs? 

•  What  information  must  an  organization  provide  to  support 
its  perception  of  intrinsic  value  of  a  CP? 

•  How  can  we  capture  the  "value  added"  by  a  CP 
implementation  at  minimal  cost  to  the  implementing  org? 

•  Are  there  specif  ic  collections  of  CPs  that  must  be 
implemented  together  in  order  for  any  of  them  to  be 
successful? 

•  Is  there  a  set  of  CPs  that  provide  value  unique  to  the  R  & 
D  community?  (The  same  set  would  not  work  well  outside 
of  R&  D) 
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Status  of  DACS  Initiative 


•  GP  Web  Site 

-  Under  development 

-  Available  in  late  Spring 

•  GP  Architecture  and  Prof  iles 

-  Initial  drafts  published  as  a  GP  Quick  Reference  on  CD  ROM 

-  Available  in  Spring 

•  Survey  is  ready 

-  Available  in  Excel  format 

-  Identifying  information  is  required 

•  DACS  is  looking  for  organizations  willing  to  develop  case 
studies 


43 


Future  DACS  Plans 


•  Partner  with  implementing  organizations  to 
develop  useful  case  studies 

•  Continue  monitoring  the  JBI  program 

-  Focusing  on  practice  interrelationships  and 

-  Evolution  of  identified  practices 

•  Identify  and  implement  other  activities  deemed 
appropriate  to  educate  the  DoD  community  and 
encourage  use  of  GPs. 

DACS  welcomes  any  dialogue  or  ideas  you  may  have! 

Please  contact  us! 
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315-330-3324 

587-3324 

Data  <&  Analysis  Center  for  Software  (DACS) 

DACS  Web  Site  [http://dacs.dtic.mil] 
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